Hopfield networks are a kind of recurrent neural network that model auto-associative memory: the ability to recall a memory from just a partial piece of that memory.
In [1]:
import os
import numpy as np
import imageio
import matplotlib
from matplotlib import pyplot as plt
from scipy.misc import imread, imsave
%matplotlib inline
matplotlib.rcParams['figure.figsize'] = (20.0, 10.0)
np.random.seed(1)
Let's load in a meme. I'm partial to 'Deal with it'.
In [2]:
deal = imread('small-deal-with-it-with-text.jpg', mode="L")
print(deal.shape)
deal = deal.astype(int)
In [3]:
np.unique(deal)
Out[3]:
To convert this to a 1 bit image, I convert everything darker than some threshold to black (1), and everything else to white (-1). Experimenting a bit with the particular image of the 'deal with it meme' that I have, a threshold of 80 seemed to work reasonably. The resulting image is still a bit rough around the edges, but it's recognizable.
In [4]:
bvw_threshold = 80
deal[deal <= bvw_threshold] = -1
deal[deal > bvw_threshold] = 1
deal = -deal
deal
Out[4]:
In [5]:
np.unique(deal)
Out[5]:
In [6]:
plt.imshow(deal, cmap='Greys', interpolation='nearest')
plt.show()
Now make weights. For now, we'll use the Hebbian learning rule, whereby two units have a positive weight (+1) if their activation is the same, and a negative weight (-1) if their activations are differnet. We also stipulate that a neuron have no weight with itself.
In [7]:
flattened_deal = deal.flatten()
flattened_deal.shape
Out[7]:
This next cell can take a little while if the image is large...
In [8]:
flatlen = len(flattened_deal)
deal_weights = np.outer(flattened_deal,flattened_deal) - np.identity(len(flattened_deal))
deal_weights[:5,:5]
Out[8]:
Now start with a noisy version of the image. We'll just flip a certain number of random pixels on each row of the image.
In [9]:
def noisify(pattern, numb_flipped=30):
noisy_pattern = pattern.copy()
for idx, row in enumerate(noisy_pattern):
choices = np.random.choice(range(len(row)), numb_flipped)
noisy_pattern[idx,choices] = -noisy_pattern[idx,choices]
return noisy_pattern
noisy_deal = noisify(pattern=deal)
In [10]:
plt.imshow(noisy_deal, cmap='Greys', interpolation='nearest')
plt.show()
Now we can start with that, and use the weights to update it. We'll update the units asynchronously (one at a time).
While we update the units, let's keep track of the energy in the network:
Updating the units' activations causes the network to move toward a local minimum in that function. Put more informally, updating the units puts the network a more 'relaxed' state.
Efficient numpy matrix math for computing the energy taken from here.
In [11]:
def flow(pattern, weights, theta=0, steps = 50000):
pattern_flat = pattern.flatten()
if (type(theta) == float) or (type(theta) == int):
thetas = np.zeros(len(pattern_flat)) + theta
for step in range(steps):
unit = np.random.randint(low=0, high=(len(pattern_flat)-1))
unit_weights = weights[unit,:]
net_input = np.dot(unit_weights,pattern_flat)
pattern_flat[unit] = 1 if (net_input > thetas[unit]) else -1
if (step % 10000) == 0:
energy = -0.5*np.dot(np.dot(pattern_flat.T,weights),pattern_flat) + np.dot(thetas,pattern_flat)
print("Energy at step {:05d} is now {}".format(step,energy))
evolved_pattern = np.reshape(a=pattern_flat, newshape=(pattern.shape[0],pattern.shape[1]))
return evolved_pattern
In [12]:
steps = 50000
theta = 0
noisy_deal_evolved = flow(noisy_deal, deal_weights, theta = theta, steps = steps)
As expected, energy decreases as the units' activations are updated. Now plot the 'evolved' pattern.
In [13]:
plt.imshow(noisy_deal_evolved, cmap='Greys', interpolation='nearest')
plt.show()
Voila.
The cooler thing about the Hopfield networks is that they can encode multiple patterns (to a limit depending on the training regimen, and the number of units). So let's try another maymay.
I got the next meme from here, and then tweaked its levels in Mac's preview so that it'd translate nicely to a 1 bit (black or white) image.
In [14]:
woah = imread('woah.png', mode="L")
woah = woah.astype(int)
woah[woah >= 1] = 1
woah[woah < 1] = -1
woah = -woah
In [15]:
np.unique(woah)
Out[15]:
In [16]:
plt.imshow(woah, cmap='Greys', interpolation='nearest')
plt.show()
Cool. So now we make some weights for this image...
In [17]:
flattened_woah = woah.flatten()
flatlen = len(flattened_woah)
woah_weights = np.outer(flattened_woah,flattened_woah) - np.identity(len(flattened_woah))
...and then just average those with the weights for the 'deal with it' network. This takes a surprisingly long time (10-15 seconds).
In [18]:
average_weights = (woah_weights + deal_weights) / 2
Now, let's make a noisy Neil deGrasse Tyson, and have the network try to recover the clean, pristine NGT.
In [19]:
noisy_woah = noisify(pattern=woah)
plt.imshow(noisy_woah, cmap='Greys', interpolation='nearest')
plt.show()
In [20]:
recovered_woah = flow(noisy_woah, average_weights, theta = theta, steps = steps)
plt.imshow(recovered_woah, cmap='Greys', interpolation='nearest')
plt.show()
Now let's doublecheck that the average weights also still work for the 'deal with it' image.
In [21]:
deal_recovered = flow(noisy_deal, average_weights, theta = theta, steps = steps)
plt.imshow(deal_recovered, cmap='Greys', interpolation='nearest')
plt.show()
Sweet. So now we can try something like feeding it a pattern that is halfway between the two patterns -- it should eventually settle into one of them! Who has greater meme strength!??!
In [22]:
deal_with_neil = (woah + deal) / 2
print(np.unique(deal_with_neil))
I could force those 0 values to -1 or 1, but that biases the pattern towards deal and neil, respectively (at least, testing suggested this -- I think because Neil has more black pixels and Deal has more white pixels). So, I'll leave them in. I could probably solve this by randomly setting 0's to 1 or -1, but naw.
In [23]:
#deal_with_neil[deal_with_neil == 0] = -1
#np.unique(deal_with_neil)
In [24]:
plt.imshow(deal_with_neil, cmap='Greys', interpolation='nearest')
plt.show()
In [25]:
recovered_deal_with_neil = flow(deal_with_neil, average_weights, theta = theta, steps = steps)
plt.imshow(recovered_deal_with_neil, cmap='Greys', interpolation='nearest')
plt.show()
Assuming the cells/pixels of 0 were unaltered, if you run that a few times, you'll notice that sometimes it settles on Neil, and sometimes it settles on Deal!!!
Hopfield networks can also settle onto 'spurious patterns' (patterns that the network wasn't trained on). For each stored pattern x, -x is a spurious pattern. But also, any linear combination of the of the learned patterns can be a spurious pattern. So let's learn a third pattern, and then see the network stabilize on a simple combination of the three patterns.
In [26]:
butt = imread('dick_butt.png', mode="L")
butt = butt.astype(int)
butt[butt >= 1] = 1
butt[butt < 1] = -1
plt.imshow(butt, cmap='Greys', interpolation='nearest')
plt.show()
In [27]:
flattened_butt = butt.flatten()
flatlen = len(flattened_butt)
butt_weights = np.outer(flattened_butt,flattened_butt) - np.identity(len(flattened_butt))
In [28]:
average_weights = (woah_weights + deal_weights + butt_weights) / 3
In [29]:
noisy_butt = noisify(pattern=butt)
plt.imshow(noisy_butt, cmap='Greys', interpolation='nearest')
plt.show()
In [30]:
recovered_butt = flow(noisy_butt, average_weights, theta=theta, steps=steps)
plt.imshow(recovered_butt, cmap='Greys', interpolation='nearest')
plt.show()
In [31]:
recovered_woah = flow(noisy_woah, average_weights, theta=theta, steps=steps)
plt.imshow(recovered_woah, cmap='Greys', interpolation='nearest')
plt.show()
Okay, now let's make a spurious pattern. Any linear combination will do.
In [32]:
spurious_meme = butt + deal + woah
np.unique(spurious_meme)
Out[32]:
In [33]:
spurious_meme[spurious_meme > 0] = 1
spurious_meme[spurious_meme < 0] = -1
In [34]:
plt.imshow(spurious_meme, cmap='Greys', interpolation='nearest')
plt.show()
Pretty noisy. Only Neal, and kiiiiinda the Deal with It, are visible. Now make a noisy version of that combination.
In [35]:
noisy_spurious_meme = noisify(pattern=spurious_meme)
plt.imshow(noisy_spurious_meme, cmap='Greys', interpolation='nearest')
plt.show()
Beautifully noisy. Can barely see anything in it. But now if we start with that, and apply the weights, it should recover the spurious pattern!
In [36]:
steps = 100000
recovered_spurious_meme = flow(noisy_spurious_meme, average_weights, theta=theta, steps=steps)
plt.imshow(recovered_spurious_meme, cmap='Greys', interpolation='nearest')
plt.show()
And it sure as heck did.
Let's make some nifty animations of the networks recovering a pattern (learned or spurious) from an input. Let's make the input more or less pure noise, by increasing numb_flipped.
In [37]:
noise = noisify(pattern=woah, numb_flipped=100)
plt.imshow(noise, cmap='Greys', interpolation='nearest')
plt.show()
Great. That sure looks like pure noise.
In [38]:
steps = 50000
recovered_meme = flow(noise, average_weights, theta=theta, steps=steps)
plt.imshow(recovered_meme, cmap='Greys', interpolation='nearest')
plt.show()
Given that the the noise is basically a random image, the network could settle on any of the attractors (modulo how attractive they each are), so the above image will vary from run to run.
In [39]:
steps = 50000
noisy_meme_flat = noise.flatten()
os.chdir('recovery_steps')
for step in range(steps):
unit = np.random.randint(low=0, high=(len(noisy_meme_flat)-1))
unit_weights = average_weights[unit,:]
net_input = np.dot(unit_weights,noisy_meme_flat)
noisy_meme_flat[unit] = 1 if (net_input > theta) else -1
recovered_meme = np.reshape(a=noisy_meme_flat, newshape=(len(butt),len(butt)))
if not (step % 1000):
imsave('recovered_meme{:05d}.jpg'.format(step), recovered_meme)
In [40]:
filenames = [x for x in os.listdir() if x[-4:] == '.jpg']
images = []
for filename in filenames:
images.append(imageio.imread(filename))
os.chdir("..")
imageio.mimsave('meme_movie.gif', images, duration=0.2)
In [ ]: